Historical and Recent Trends in the Forecasting Literature

Talk for the WARN-D Friday Science Meeting

February 16, 2024

Introduction

Why talk about this topic?

the secret reason:

Ok but really, why should you care?

“The quiet revolution of numerical weather prediction” (Bauer, Thorpe, and Brunet 2015)

Forecasting Competitions

History

Critical statisticians

  • We should be looking for the true model!

  • Maybe you do not know how to model with ARIMA…

  • I suspect it is more likely to depend on the skill of the analyst … these authors are more at home with simple procedures than with Box-Jenkins. (Chatfield)

Makridakis & Hibon (1979)

  • Our Empirical evidence disagrees

  • Of course we do

  • might be useful for Dr. Chatfield to read some of the psychological literature quoted in the main paper, and he can then learn a little more about biases

M-Competitions

Competition Year N° Time Series Insights/Novelty
M1 1982 1001
  • Easy methods work well

  • Combining methods works well

  • Changed forecasting forever

M2 1993 29
  • Not really relevant
M3 2000 3003
  • Somewhat simple models work well with modifications
M4 2020 100,000
  • Combination of ML and Stats works well, pure ML not

  • Probabilistic Prediction

M5 2021 42,000
  • Hierarchical Time Series

  • Ensembles + Pure ML works well

M6 2022 100
  • … to be continued

Utility of Competitions

Advantages

  • Empirical evidence
  • Benchmarking
  • Methodological development and cumulative science (Fildes and Ord 2004)

Issues

Model Selection, Uncertainty & Combination

Model selection

The beauty of simple models:

  • Simple models often outperform more complex ones, even on larger data sets (M1, M3)

  • Simple models are important benchmarks, even in more complex settings

Selecting a single model:

  • ignores uncertainty about this selection (Kaplan 2021)
  • tends to perform poorly in forecasting competitions

Model combination

‘Law’ of forecast combination

‘The results have been virtually unanimous: combining multiple forecasts leads to increased forecast accuracy. In many cases one can make dramatic performance improvements by simply averaging the forecasts.’ (Clemen 1989)

Important:

Ensemble modeling

Two sources of uncertainty:

Met Office UK & Bauer, Thorpe, and Brunet (2015)

Relevance for psychology

  • Model uncertainty often neglected, both in inferential and predictive modeling (Kaplan 2021)
  • Theoretically linked to the complex systems literature
  • Helps us understand which methods work when

Probabilistic Forecasting

Probabilistic forecasting methods

What is probabilistic forecasting?

  • what is even meant here?
  • show plot by gneiting in the appendix
  • show different forms

Moving towards probabilistic forecasting

Long history of probabilistic weather forecasting

The probability of rain was much smaller than at other times (Dalton, 1793)

  • Popularized by Epstein 1969 Stochastic dynamic prediction

Some more stuff on probabilistic forecasting

Relevance for psychology

  • Practically linked to decision theory, e.g., for JITAIs

Mixed Models

Extending mixed models

Flexible Mixed Model

From

\[ y = X\beta + Z\upsilon + \epsilon \]

to

\[y = ml_{fixed}(X)+Z\upsilon + \epsilon\]

(Kilian, Ye, and Kelava 2023)

  • not ‘forecasting’ literature, but ML more broadly
  • use of random effects in machine learning has gained attention

Relevance for Psychology

  • Current combinations of ‘idiographic’ and ‘nomethetic’ forecasts: often ad-hoc
  • Improving on what we already have

Summary

Issues

Methodology

Transfer to Psychology

Takeaways

  1. Forecasting competitions can lead to methodological improvement
  2. Model uncertainty and model combination are integral parts of forecasting
  3. Probabilistic forecasting is important for decision making
  4. Combining predictions is challenging, but lots of cool stuff in the making

Me

Feel

free

to

contact

me

bjoern.siepe@uni-marburg.de

References

Aastveit, Knut Are, James Mitchell, Francesco Ravazzolo, and Herman van Dijk. n.d. “The Evolution of Forecast Density Combinations in Economics.”
Bauer, Peter, Alan Thorpe, and Gilbert Brunet. 2015. “The Quiet Revolution of Numerical Weather Prediction.” Nature 525 (7567, 7567): 47–55. https://doi.org/10.1038/nature14956.
Boylan, John E., Paul Goodwin, Maryam Mohammadipour, and Aris A. Syntetos. 2015. “Reproducibility in Forecasting Research.” International Journal of Forecasting 31 (1): 79–90. https://doi.org/10.1016/j.ijforecast.2014.05.008.
Clemen, T. 1989. “Combining Forecasts: A Review and Annotated Bibliography.” International Journal of Forecasting 5 (4): 559–83.
Fildes, Robert, and Keith Ord. 2004. “Forecasting Competitions: Their Role in Improving Forecasting Practice and Research.” In, edited by Michael P. Clements and David F. Hendry, 1st ed. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470996430.
Kaplan, David. 2021. “On the Quantification of Model Uncertainty: A Bayesian Perspective.” Psychometrika 86 (1): 215–38. https://doi.org/10.1007/s11336-021-09754-5.
Kilian, Pascal, Sangbeak Ye, and Augustin Kelava. 2023. “Mixed Effects in Machine Learning – A Flexible mixedML Framework to Add Random Effects to Supervised Machine Learning Regression.” Transactions on Machine Learning Research.
Koning, Alex J., Philip Hans Franses, Michèle Hibon, and H. O. Stekler. 2005. “The M3 Competition: Statistical Tests of the Results.” International Journal of Forecasting 21 (3): 397–409. https://doi.org/10.1016/j.ijforecast.2004.10.003.
Makridakis, Spyros, Rob J. Hyndman, and Fotios Petropoulos. 2020. “Forecasting in Social Settings: The State of the Art.” International Journal of Forecasting, M4 Competition, 36 (1): 15–28. https://doi.org/10.1016/j.ijforecast.2019.05.011.
Murphy, Allan H. 1998. “The Early History of Probability Forecasts: Some Extensions and Clarifications.” Weather and Forecasting 13 (1): 5–15. https://doi.org/10.1175/1520-0434(1998)013<0005:TEHOPF>2.0.CO;2.
Strobl, Carolin, and Friedrich Leisch. 2022. “Against the ‘One Method Fits All Data Sets’ Philosophy for Comparison Studies in Methodological Research.” Biometrical Journal Advance Online Publication. https://doi.org/10.1002/bimj.202200104.
Wang, Xiaoqian, Rob J. Hyndman, Feng Li, and Yanfei Kang. 2023. “Forecast Combinations: An over 50-Year Review.” International Journal of Forecasting 39 (4): 1518–47. https://doi.org/10.1016/j.ijforecast.2022.11.005.